vtarcfandomcom-20200213-history
Application Software Installation
Write the first paragraph of your page here. Overview ARC uses the rpmbuild '''tool to manage the applications that are installed on our systems. This document outlines the process required to build a typical application within this framework. When building applications, ARC systems utilize a build node where compilation and testing can take place prior to the installation of an application on the production system. These build nodes can be accessed as follows: On blueridge: $ ssh brbuild On hokiespeed: $ ssh hsbuild On hokieone: ssh hobuild The ultimate installation path for a particular application is determined based on the compiler and MPI stack with which it was compiled. Rather than working with standard rpmbuild tool, members of the '''appsadm group rely on a wrapper which sets many of the options. The permissible options for the wrapper are available from: $ rpmbuild -h Details for the functionality of the ARC-specific rpmbuild wrapper are available by viewing the associated script: $ more /usr/local/bin/rpmbuild Initial Setup Configure git: Whenever you make a commit to the git repository it will automatically add your name and email address. This information is typically provided to git as a one-time setup step by running the following commands $ git config --global user.name "Your Name" $ git config --global user.email "you@domain.name" ARC mainains a git repository of existing spec files that can be updatedy by ARC staff members. Access to the repository is managed by ssh, and your public ssh key must be added to the repository by UAS staff (Chris Snapp). Once your ssh key pairs have been configured, you will be able to clone the repository as follows: $ git clone git@uaserve.cc.vt.edu:arc/specs.git Creating the Spec File Before one can build a rpm we need a spec file which describes how it is built. Figure~\ref{fig:spec_file} presents the anatomy of a spec file. We can consider the spec file to be divided into four sections #Description: information about the application being built, version number, one-line description #Setup: unpack the files that are associated with the application and patch as necessary #Build: the build phase, including loading of modules, compiling the package, and finally generating a new module file associated with a particular application #Packaging: package the required files and module file into an RPM Many lines will remain essentially unchanged from spec file to spec file. Moreover, there are now many spec files in the repository, from the very simple ``just type `\verb|make|'" files like bowtie2 to spec files to add to already existing modules like numpy. So when you are building a software package, think about what other software might be built in a similar way and use that spec file as a reference. ''Note: If you are modifying an existing spec file, make sure to bump the release number in the description part of the spec file.'' Building an RPM Once the spec file is ready we build the rpm. To build the rpm we use the following command: $ rpmbuild foo.spec If applicable, you will be prompted for a choice of compiler and MPI stack and asked to hit `y' to continue once or twice. At the end, rpmbuild will report either SUCCESS or FAILURE. There is a lot of information that flies through, if one wants to create a build log and monitor the build process simultaneously we can modify the command as follows: $ rpmbuild foo.spec 2>&1 | tee /path/to/log/file One can then inspect the build log file and check for any build errors or better understand the build error. If everything goes well one should have the rpm built successfully. Once an RPM has been created successfully and tested (see below) then it can be built against all combinations of compilers and MPI stacks using: $ rpmbuild -a foo.spec Testing an RPM The next stage after building the rpm is testing that everything is fine. To do so, one can extract the files to our home directory for testing. To extract the rpm to the current directory run the following command: rpm2cpio /path/to/rpm/file.rpm | cpio -idv After the files are extracted one needs to modify the associated module file to reflect the path changes. To use the modified module file for testing we need to modify the MODULEPATH so that we can load it. This can be done by appending or prepending the location of the module file(s) to the MODULEPATH environment variable as follows: $ export MODULEPATH=$MODULEPATH:/path/to/extracted/modulefile/directory $ module avail Now one can test the application by loading the module. Note, however, that any environment variables set in the module will still point to the final location (where files will be when the module is installed into the apps stack), whereas the unpacked files will be in a local path. So rather than using, for example: ls $FOO_DIR to view the contents of the folder specified by the |FOO_DIR| environment variable, you will simply need to use: ls .$FOO_DIR| (note the period specifying that the path is to begin in the current directory). Installing the RPM The modules in the apps stack have not been updated until the RPM has been installed. A member of the UAS team will ultimately be responsible for the installation of RPMs on ARC systems. At some point, the 'appsadm' group may get access to the rpms. In that case, they can be manipulated as follows: Erase a particular rpm: sudo rpm -e /opt/build/RPMS/x86_64/vt-foo-rpm-etc List all `foo' rpms: rpm -qa | grep foo Install a particular rpm: sudo rpm -ivh vt-foo-rpm-etc.rpm Upload to the git Repository Once the rpm has been successfully built, we need to upload the spec file to the git repository. This is extremely important so that (1) everyone has access to it so the software can be built on other systems later and (2) so that the latest, working version is in the repository for people to use if it needs to be tweaked/updated later. To do this: #Use '''git status' '' to show how your local copy of the repository differs from the remote repository. If you have created a new spec file, you should see that file listed under 'Untracked Files'. #If you have a new spec file to ad to the repository, run 'git add foo.spec' to add it to the staging area. Note that this does not upload it to the repository. #Run 'git pull' to ensure that you have the latest version of all other files from the remote repository #Run 'git commit -m 'sys: message' to commit files that have been staged. 'sys' should be replaced by br, hs, etc., depending on the particular system. The message should describe briefly the changes that have been made. #Run 'git push' to push your changes to the remote repository. If your key requires a password you will be prompted to enter it. Once these steps have been completed, you are done building your RPM.